Collaborative learning framework for online stakeholder engagement.

BACKGROUND: Public and stakeholder engagement can improve the quality of both research and policy decision making. However, such engagement poses significant methodological challenges in terms of collecting and analysing input from large, diverse groups. OBJECTIVE: To explain how online approaches can facilitate iterative stakeholder engagement, to describe how input from large and diverse stakeholder groups can be analysed and to propose a collaborative learning framework (CLF) to interpret stakeholder engagement results. METHODS: We use 'A National Conversation on Reducing the Burden of Suicide in the United States' as a case study of online stakeholder engagement and employ a Bayesian data modelling approach to develop a CLF. RESULTS: Our data modelling results identified six distinct stakeholder clusters that varied in the degree of individual articulation and group agreement and exhibited one of the three learning styles: learning towards consensus, learning by contrast and groupthink. Learning by contrast was the most common, or dominant, learning style in this study. CONCLUSION: Study results were used to develop a CLF, which helps explore multitude of stakeholder perspectives; identifies clusters of participants with similar shifts in beliefs; offers an empirically derived indicator of engagement quality; and helps determine the dominant learning style. The ability to detect learning by contrast helps illustrate differences in stakeholder perspectives, which may help policymakers, including Patient-Centered Outcomes Research Institute, make better decisions by soliciting and incorporating input from patients, caregivers, health-care providers and researchers. Study results have important implications for soliciting and incorporating input from stakeholders with different interests and perspectives.

Reducing the burden of suicide in the U.S.: the aspirational research goals of the National Action Alliance for Suicide Prevention Research Prioritization Task Force.

BACKGROUND: The National Action Alliance for Suicide Prevention Research Prioritization Task Force (RPTF) has created a prioritized national research agenda with the potential to rapidly and substantially reduce the suicide burden in the U.S. if fully funded and implemented. PURPOSE: Viable, sustainable scientific research agendas addressing challenging public health issues such as suicide often need to incorporate perspectives from multiple stakeholder groups (e.g., researchers, policymakers, and other end-users of new knowledge) during an agenda-setting process. The Stakeholder Survey was a web-based survey conducted and analyzed in 2011-2012 to inform the goal-setting step in the RPTF agenda development process. The survey process, and the final list of "aspirational" research goals it produced, are presented here. METHODS: Using a modified Delphi process, diverse constituent groups generated and evaluated candidate research goals addressing pressing suicide prevention research needs. RESULTS: A total of 716 respondents representing 49 U.S. states and 18 foreign countries provided input that ultimately produced 12 overarching, research-informed aspirational goals aimed at reducing the U.S. suicide burden. Highest-rated goals addressed prevention of subsequent suicidal behavior after an initial attempt, strategies to retain patients in care, improved healthcare provider training, and generating care models that would ensure accessible treatment. CONCLUSIONS: The Stakeholder Survey yielded widely valued research targets. Findings were diverse in focus, type, and current phase of research development but tended to prioritize practical solutions over theoretical advancement. Other complex public health problems requiring input from a broad-based constituency might benefit from web-based tools that facilitate such community input.

How can we recognize continuous quality improvement?

OBJECTIVE: Continuous quality improvement (CQI) methods are foundational approaches to improving healthcare delivery. Publications using the term CQI, however, are methodologically heterogeneous, and labels other than CQI are used to signify relevant approaches. Standards for identifying the use of CQI based on its key methodological features could enable more effective learning across quality improvement (QI) efforts. The objective was to identify essential methodological features for recognizing CQI. DESIGN: Previous work with a 12-member international expert panel identified reliably abstracted CQI methodological features. We tested which features met rigorous a priori standards as essential features of CQI using a three-phase online modified-Delphi process. SETTING: Primarily United States and Canada. PARTICIPANTS: 119 QI experts randomly assigned into four on-line panels. INTERVENTION: Participants rated CQI features and discussed their answers using online, anonymous and asynchronous discussion boards. We analyzed ratings quantitatively and discussion threads qualitatively. Main outcome measure(s) Panel consensus on definitional CQI features. RESULTS: /st> Seventy-nine (66%) panelists completed the process. Thirty-three completers self-identified as QI researchers, 18 as QI practitioners and 28 as both equally. The features 'systematic data guided activities,' 'designing with local conditions in mind' and 'iterative development and testing' met a priori standards as essential CQI features. Qualitative analyses showed cross-cutting themes focused on differences between QI and CQI. CONCLUSIONS: We found consensus among a broad group of CQI researchers and practitioners on three features as essential for identifying QI work more specifically as 'CQI.' All three features are needed as a minimum standard for recognizing CQI methods.

A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating.

BACKGROUND: Comparative effectiveness and systematic reviews require frequent and time-consuming updating. RESULTS: of earlier screening should be useful in reducing the effort needed to screen relevant articles. METHODS: We collected 16,707 PubMed citation classification decisions from 2 comparative effectiveness reviews: interventions to prevent fractures in low bone density (LBD) and off-label uses of atypical antipsychotic drugs (AAP). We used previously written search strategies to guide extraction of a limited number of explanatory variables pertaining to the intervention, outcome, and STUDY DESIGN: We empirically derived statistical models (based on a sparse generalized linear model with convex penalties [GLMnet] and a gradient boosting machine [GBM]) that predicted article relevance. We evaluated model sensitivity, positive predictive value (PPV), and screening workload reductions using 11,003 PubMed citations retrieved for the LBD and AAP updates. Results. GLMnet-based models performed slightly better than GBM-based models. When attempting to maximize sensitivity for all relevant articles, GLMnet-based models achieved high sensitivities (0.99 and 1.0 for AAP and LBD, respectively) while reducing projected screening by 55.4% and 63.2%. The GLMnet-based model yielded sensitivities of 0.921 and 0.905 and PPVs of 0.185 and 0.102 when predicting articles relevant to the AAP and LBD efficacy/effectiveness analyses, respectively (using a threshold of P ≥ 0.02). GLMnet performed better when identifying adverse effect relevant articles for the AAP review (sensitivity = 0.981) than for the LBD review (0.685). The system currently requires MEDLINE-indexed articles. CONCLUSIONS: We evaluated statistical classifiers that used previous classification decisions and explanatory variables derived from MEDLINE indexing terms to predict inclusion decisions. This pilot system reduced workload associated with screening 2 simulated comparative effectiveness review updates by more than 50% with minimal loss of relevant articles.

Conducting online expert panels: a feasibility and experimental replicability study.

BACKGROUND: This paper has two goals. First, we explore the feasibility of conducting online expert panels to facilitate consensus finding among a large number of geographically distributed stakeholders. Second, we test the replicability of panel findings across four panels of different size. METHOD: We engaged 119 panelists in an iterative process to identify definitional features of Continuous Quality Improvement (CQI). We conducted four parallel online panels of different size through three one-week phases by using the RAND's ExpertLens process. In Phase I, participants rated potentially definitional CQI features. In Phase II, they discussed rating results online, using asynchronous, anonymous discussion boards. In Phase III, panelists re-rated Phase I features and reported on their experiences as participants. RESULTS: 66% of invited experts participated in all three phases. 62% of Phase I participants contributed to Phase II discussions and 87% of them completed Phase III. Panel disagreement, measured by the mean absolute deviation from the median (MAD-M), decreased after group feedback and discussion in 36 out of 43 judgments about CQI features. Agreement between the four panels after Phase III was fair (four-way kappa=0.36); they agreed on the status of five out of eleven CQI features. Results of the post-completion survey suggest that participants were generally satisfied with the online process. Compared to participants in smaller panels, those in larger panels were more likely to agree that they had debated each others' view points. CONCLUSION: It is feasible to conduct online expert panels intended to facilitate consensus finding among geographically distributed participants. The online approach may be practical for engaging large and diverse groups of stakeholders around a range of health services research topics and can help conduct multiple parallel panels to test for the reproducibility of panel conclusions.

Using information mining of the medical literature to improve drug safety.

OBJECTIVE: Prescription drugs can be associated with adverse effects (AEs) that are unrecognized despite evidence in the medical literature, as shown by rofecoxib's late recall in 2004. We assessed whether applying information mining to PubMed could reveal major drug-AE associations if articles testing whether drugs cause AEs are over-represented in the literature. DESIGN: MEDLINE citations published between 1949 and September 2009 were retrieved if they mentioned one of 38 drugs and one of 55 AEs. A statistical document classifier (using MeSH index terms) was constructed to remove irrelevant articles unlikely to test whether a drug caused an AE. The remaining relevant articles were analyzed using a disproportionality analysis that identified drug-AE associations (signals of disproportionate reporting) using step-up procedures developed to control the familywise type I error rate. MEASUREMENTS: Sensitivity and positive predictive value (PPV) for empirical drug-AE associations as judged against drug-AE associations subject to FDA warnings. RESULTS: In testing, the statistical document classifier identified relevant articles with 81% sensitivity and 87% PPV. Using data filtered by the statistical document classifier, base-case models showed 64.9% sensitivity and 42.4% PPV for detecting FDA warnings. Base-case models discovered 54% of all detected FDA warnings using literature published before warnings. For example, the rofecoxib-heart disease association was evident using literature published before 2002. Analyses incorporating literature mentioning AEs common to the drug class of interest yielded 71.4% sensitivity and 40.7% PPV. CONCLUSIONS: Results from large-scale literature retrieval and analysis (literature mining) compared favorably with and could complement current drug safety methods.